Integration of shape from shading and stereo

-Stereo algorithms suffer from the lack of local surface texture due to smoothness of depth constraint, or local miss-matches in disparity estimates. Thus, most stereo methods only provide a coarse depth map which can be associated with a low pass image of the depth map. On the other hand, shape from shading algorithms generally produce better estimates of local surface areas, but some of them have problems with variable albedo and spherical surfaces. Thus, shape from shading methods produce better detailed depth information, and can be associated with the high pass image of the depth map. In order to compute a better depth map, we present a method for integrating the high frequncy information from the shape from shading and the low frequency information from stereo. The proposed algorithm is very simple, takes about 0.7 s for a 128 x 128 image on a Sun SparcStation-1, is non-iterative, and requires very little adjustment of parameters. The results obtained with a variety of synthetic and real images are discussed. The quality of depth obtained by integrating shading and stereo is compared with the ground truth (range image) using height error measure, and improvement ranging from 30 to 50% over stereo, and from 65 to 98% over shading is demonstrated. Shape from shading Shape from stereo Human Visual System Integration of visual modules l . I N T R O D U C T I O N Modern Computer Vision research follows the Marr paradigm I1) that treats vision as a large, complex information processing systems. Individual perceptual modules can be identified in the system which are responsible for the computat ion of shape from shading, stereo, motion, texture and contour, as well as processes for determining the location and nature of illumination sources, three dimensional motion, and other methods. During the last two decades • there has been significant interest in these individual modules, which are termed shape from X. Interesting results have been reported, in particular, in motion, stereo and shading. Marr envisioned that the output from the individual modules will ultimately be integrated into a single representation called 2.5 D sketch. However, this integration was never accomplished by Marr. Vision inherently is an illposed problem and the solutions for shape from X obtained by considering each module individually may not necessarily exist, may not be unique and may not be stable. Therefore, in order to tackle these problems we need more information. In particular, if we combine information from different image cues like stereo, shading and motion, the solution may be significantly improved. Surprisingly, it is only recently that researchers in Computer Vision have started realizing the benefits * The research reported here was supported by the National Science Foundation under grant number CDA 9200369. -t Author to whom correspondence should be addressed. of integrating information from separate modules. these are the work of Horn c2) on combining shading with contour, Grimson's c3) use of shading in determining the surface orientation of feature-point contours obtained from stereo, Aloimonos 's methods c4~ for combining shading and motion, texture and motion, and mot ion and contour, and Waxman's approach for combining stereo and motion. (s~ The objective of this research is to work on the integration of shape from X modules. In particular, we are interested in combining the depth information (3D shape) from two very important cues, stereo and shading. Shape from shading is the estimation of 3D shape from a 2D image given the light source and surface reflectance information. Stereo is the estimation of 3D shape from two images taken by cameras which are slightly shifted along the axis which the two cameras are aligned on (usually the x axis). The image of the object in the left stereo image is shifted with respect to the image in the right stereo image. This shift, which is also called the disparity, is inversely proport ional to the 3D distance (depth) from the camera to the object. Frankot and Cheltappa c6~ pointed out that correspondence between stereo image pairs provides low frequency information not available in shading alone, and shading provides information not available from either sparse or low resolution stereo correspondences. Pentland (7~ also suggested that linear shape from shading would be most useful in conjunction with some other depth cue, such as stereo, which could reliably provide the coarse, low,frequency structure of the